Papers with fine-tuning loss
The Craft of Selective Prediction: Towards Reliable Case Outcome Classification - An Empirical Study on European Court of Human Rights Cases (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing COC tasks prioritize high task performance over model reliability . however, large models exhibit overconfidence and Monte Carlo dropout methods produce reliable confidence estimates . |
| Approach: | They conduct an empirical investigation into how various design choices affect the reliability of COC models within the framework of selective prediction. |
| Outcome: | The proposed model is able to predict the outcome of a legal case based on the text of the case facts and is compared with other models using a pre-training corpus. |
Uncertainty Quantification with Pre-trained Language Models: A Large-Scale Empirical Analysis (2022.findings-emnlp)
Copied to clipboard
Yuxin Xiao, Paul Pu Liang, Umang Bhatt, Willie Neiswanger, Ruslan Salakhutdinov, Louis-Philippe Morency
| Challenge: | Pre-trained language models (PLMs) have gained increasing popularity due to compelling prediction performance in diverse natural language processing tasks. |
| Approach: | They compare three popular options for encoding and Temp Scaling for PLMs . they recommend using Temp Loss as uncertainty quantifier and Focal Loss for fine-tuning . |
| Outcome: | Using pre-trained language models, we compare three options on NLP classification tasks and domain shift. |
The Role of Outgoing Connection Heterogeneity in Feedforward Layers of Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Using a new fine-tuning loss, we show that inner neurons with diverse outgoing connections are more critical to model performance than those with uniform connections. |
| Approach: | They propose a new loss that reduces the outgoing connection entropy in feedforward layers and elucidates the role of outgoing connections in large language models. |
| Outcome: | The proposed method is significantly more effective than removing neurons randomly or based on their magnitude. |